Live-assisted image acquisition method and system with charged particle microscopy

ABSTRACT

A method of imaging a sample includes acquiring one or more first images of a region of the sample at a first imaging condition with a charged particle microscope system. The one or more first images are applied to an input of a trained machine learning model to obtain a predicted image indicating atom structure probability in the region of the sample. An enhanced image indicating atom locations in the region of the sample based on the atom structure probability in the predicted image is caused to be displayed in response to obtaining the predicted image.

FIELD

The field relates to charged particle microscopy.

BACKGROUND

Charged particle microscopy involves using a beam of accelerated chargedparticles as a source of illumination. Example types of charged particlemicroscopy include transmission electron microscopy, scanning electronmicroscopy, scanning transmission electron microscopy, and ion beammicroscopy.

Scanning transmission electron microscopy, also known as STEM, can beused to acquire high resolution images of samples at an atomic scale. InSTEM imaging, an electron beam is scanned over a sample or an area ofthe sample. The electrons interact with the sample, resulting inelastically scattered electrons exiting the sample. In a transmissionimaging mode, the electrons transmitted through the sample are detectedand used to form a microscopic image of the sample.

STEM atomic spectroscopy can give insight into the structure of amaterial. However, obtaining atomic images with sufficient resolutionfor material structure analysis can be challenging. In a typical atomicspectroscopy workflow, an operator navigates around the sample to findinteresting areas on the sample. When the interesting areas are found, afinal image of the interesting areas is taken. The navigation involvesscanning an electron beam across the sample, or areas of the sample. Toobtain atomic resolution images, the electron beam should be alignedwith the atom columns in the irradiated areas of the sample, which mayrequire tilting of the sample relative to the electron beam path to findthe desired alignment. Many materials of interest have a limitedelectron dose budget. For these materials, a lengthy navigation to findinteresting areas and/or navigation using a high radiation dose (e.g.,100 e⁻/Å²) can result in damage to the sample prior to acquiring thefinal image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic of a system for sample imaging with chargedparticle microscopy, in accordance with one implementation.

FIG. 2 is a block diagram of an example sample imaging application,according to one implementation.

FIG. 3A illustrates a low signal-to-noise (SNR) image acquired with acharged particle microscope system.

FIG. 3B illustrates a predicted image generated by a trained machinelearning network based on the low SNR image of FIG. 3A.

FIG. 3C illustrates an enhanced image generated based on the predictedimage of FIG. 3B.

FIG. 4A is a flow diagram illustrating a method of navigating around asample with charged particle microscopy.

FIG. 4B is a flow diagram illustrating a portion of the method depictedin FIG. 4A.

FIG. 5 is an example graphical user interface that may be presented bythe sample imaging application depicted in FIG. 2 , according to oneimplementation.

FIG. 6 is an illustration of an implementation of a cycle generativeadversarial network (CycleGAN).

DETAILED DESCRIPTION Example—General Considerations

The subject matter is described with implementations and examples. Insome cases, as will be recognized by one skilled in the art, thedisclosed implementations and examples may be practiced without one ormore of the disclosed specific details, or may be practiced with othermethods, structures, and materials not specifically disclosed herein.All the implementations and examples described herein and shown in thedrawings may be combined without any restrictions to form any number ofcombinations, unless the context clearly dictates otherwise, such as ifthe proposed combination involves elements that are incompatible ormutually exclusive. The sequential order of the acts in any processdescribed herein may be rearranged, unless the context clearly dictatesotherwise, such as if one act requires the result of another act asinput.

In the interest of conciseness, and for the sake of continuity in thedescription, same or similar reference characters may be used for sameor similar elements in different figures, and description of an elementin one figure will be deemed to carry over when the element appears inother figures with the same or similar reference character. In somecases, the term “corresponding to” may be used to describecorrespondence between elements of different figures. In an exampleusage, when an element in a first figure is described as correspondingto another element in a second figure, the element in the first figureis deemed to have the characteristics of the other element in the secondfigure, and vice versa, unless stated otherwise.

The word “comprise” and derivatives thereof, such as “comprises” and“comprising”, are to be construed in an open, inclusive sense, that is,as “including, but not limited to”. The singular forms “a”, “an”, “atleast one”, and “the” include plural referents, unless the contextdictates otherwise. The term “and/or”, when used between the last twoelements of a list of elements, means any one or more of the listedelements. The term “or” is generally employed in its broadest sense,that is, as meaning “and/or”, unless the context clearly dictatesotherwise. When used to describe a range of dimensions, the phrase“between X and Y” represents a range that includes X and Y. As usedherein, an “apparatus” may refer to any individual device, collection ofdevices, part of a device, or collections of parts of devices.

Example—Overview

The subject matter disclosed herein pertains to imaging of a sample withcharged particle microscopy under imaging conditions that can result inacquired images with low signal-to-noise ratio (SNR). In these low SNRimages, the atom structures in the imaged area of the sample are notclearly visible to the naked eye because of the low resolution of theimages. The methods and systems disclosed herein can use low SNR imagesacquired with charged particle microscopy to generate enhanced imagesthat indicate information about the atomic structure in the imaged areaof the sample. The enhanced images can be displayed in a user interfacein order to provide live assistance to an operator while navigating onthe sample to find critical areas (e.g., inhomogeneous structure) of thesample or during in-situ experiments to capture dynamic effects insample composition.

In one implementation, the methods and systems disclosed herein can beconfigured to obtain image data of a region of the sample, generate oneor more initial images from the image data, and generate one or moreenhanced images that indicate information about the atomic structure inthe region of the sample. In various examples, the generation of the oneor more enhanced images includes predicting atom structure probabilityin the sample using a trained machine learning model that accepts theinitial image(s) as input. The predicted atom structure probability canbe used to detect atom locations in the region of the sample.

In various examples, the imaging conditions that result in acquiredimages with low SNR can include a low dose of the charged particle beam,short dwell time (or fast scan speed) of the charged particle beam, orsparse scanning. Thus, the methods and systems disclosed herein canenable live-assisted low charged particle beam dose and/or fast scanningand/or sparse scanning of a sample. In some examples, the methods andsystems can enable imaging of the sample using a charged particle beamdose that is significantly lower than a dose required to acquire a highresolution image of the sample. Advantageously, an area of the samplecan be repeatedly scanned with the low dose during the live assistancebefore the accumulated irradiation with the low dose will be equivalentto the irradiation with a single high dose scan. The low dose can alsohelp avoid a scenario where the charged particle beam alters thedynamics of the structures in the sample while capturing an image of thesample with live assistance.

Example—Sample Imaging System with Live Assistance

FIG. 1 illustrates one implementation of a system 100 for sample imagingwith live assistance. The system 100 includes a charged particlemicroscope system 104 for investigation and analysis of a sample. Thesystem 100 can further include a computing environment 116 with a sampleimaging application 200. In the illustrated example, the chargedparticle microscope system 104 includes a charged particle microscope108, enclosed within a vacuum chamber 120, and a controller 112, whichcan be within or outside the vacuum chamber 120. The controller 112 iscommunicatively coupled to the computing environment 116.

The controller 112 is connected to various components of the chargedparticle microscope 108 and can communicate control/power signals to thecomponents as well as receive data from the components. The controller112 can allow control of operations of the charged particle microscopesystem 104 from the sample imaging application 200 as well as allowimage data acquired by the charged particle microscope 108 to betransmitted to the computing environment 116 and used by the sampleimaging application 200. The controller 112 can be implemented with anysuitable combination of hardware and software.

Any type of charged particle microscope 108 suitable for acquiringimages of a sample can be used in the system 100. For illustrationpurposes, the charged particle microscope 108 is depicted as a scanningtransmission electron microscope (STEM), but the charged particlemicroscope 108 is not limited to the particular STEM configurationdepicted. The STEM can be operated in a STEM mode (i.e., where thecharged particle beam is scanned over an area of the sample) or in atransmission electron microscope (TEM) mode (i.e., without scanning thebeam). Other examples of microscopes can include, but are not limitedto, cryo-electron microscope (cryo-EM), ion based microscope, and protonmicroscope.

The charged particle microscope 108 includes a first electro-opticalsystem 144 that defines an optical axis 124. A sample 128 to beinvestigated and/or analyzed can be positioned below the firstelectro-optical system 144 and along the optical axis 124. The sample128 is supported by a sample holder 132 (or stage), which in someexamples can have capabilities to translate, rotate, and/or tilt thesample. The controller 112 can be connected to the sample holder 132 toprovide sample position control signals to the sample holder 132. Thesample holder 132 can allow different areas of the sample to bepositioned and/or tilted relative to the optical axis 124 (e.g., duringnavigation on the sample to find areas of interest on the sample).

The charged particle microscope 108 includes a charged particle source136 positioned above the first electro-optical system 144. The chargedparticle source 136 can be, for example, an electron source (e.g., aSchottky gun), a positive ion source (e.g., a gallium ion source or ahelium ion source), a negative ion source, a proton source, or apositron source. The charged particle source 136 produces a chargedparticle beam 140. The first electro-optical system 144 receives thecharged particle beam 140 and configures the charged particle beam 140into a field of view on the sample 128.

The first electro-optical system 144 can include one or moreelectro-optical components. In some examples, the electro-opticalcomponents can be connected to the controller 112 to allow theelectro-optical power of the components to be set and adjusted by thecontroller 112. For illustrative purposes, the first electro-opticalsystem 144 can include condenser lenses 148 a, 148 b, condenserstigmator 148 c, and condenser aperture 148 d. The first electro-opticalsystem 144 can include scan coils 150, which can be operated to scan thecharged particle beam 140 over an area of the sample 128. In otherexamples, the first electro-optical system 144 can have other componentsor fewer components than illustrated.

When the charged particle beam 140 is incident on the sample 128, thecharged particles in the beam interact with the structures (e.g., atoms)in the sample 132 in such a way as to cause various types of radiationto emanate from the sample 132. For example, when the charged particlebeam 132 is an electron beam, the radiation that emanates from thesample can include any combination of Auger electrons, secondaryelectrons, X-rays, backscatter electrons, cathodoluminescence, losselectrons, transmitted electrons, and diffracted electrons. The varioustypes of radiation can be detected and used to form one or more imagesof the sample.

In one implementation, the charged particle microscope 108 can includeone or more detector systems to capture images of the sample 128 withone or more detector modalities. In one example, the charged particlemicroscope 108 can include a first detector system 152 to acquire imagedata from the sample 128. The first detector system 152 can include aSTEM detector, such as a bright field detector, an annular bright fielddetector, a dark field detector, an annular dark field detector, ahigh-angle annular dark field (HAADF) detector, a segmented STEMdetector, or an integrated differential phase contrast (iDPC) detector.In one specific example, the first detector system 152 can include aHAADF detector, which detects charged particles (e.g., electrons) thatare scattered to higher angles. In another specific example, the firstdetector system 152 can include an annular bright field detector and anannular dark field detector to capture bright field and dark fieldimages of the sample simultaneously.

In one example, the charged particle microscope 108 can include one ormore additional detector systems, for example, a second detector system156 and a third detector system 160, to obtain additional image datafrom the sample. For example, the detector systems 156, 160 can bespectroscopy systems. For illustrative purposes, the second detectorsystem 156 is shown positioned below the sample 128, whereas the thirddetector system 160 is shown positioned above the sample 128. In onespecific example, the second detector system 156 can include an electronenergy loss spectroscopy, and the third detector system 160 can includean energy dispersive X-ray spectroscopy.

The charged particle microscope 108 can include a second optical system164 to direct the charged particles transmitted through the sample 128into the fields of view of the detector systems 152, 156. Forillustrative purposes, the second optical system 164 can include anobjective lens 168 a, an objective stigmator 168 b, an objectiveaperture 168 c. In other examples, the second optical system 164 canhave other components or fewer components than depicted.

The computing environment 116 can have any suitable configuration to runthe sample imaging application 200. For example, the computingenvironment 116 can include a processor 180, memory 184, a displaydevice 188, and a data storage 192. The sample imaging application 200can be loaded into memory 184 and executed by the processor 180. Thesample imaging application 200 can present a user interface on thedisplay device 188 and can present sample images in the user interfaceas well as collect microscope control settings from the user interface.The sample imaging application 200 can provide the microscope controlsettings to the controller 112 and receive detector data from thecontroller 112. The computing environment 116 can include othercomponents not specifically shown, such as input device(s), other outputdevice(s) besides the display device 188, communication connection(s),other memory besides memory 184, and other processors besides processor180.

Example—Sample Imaging Application

FIG. 2 illustrates one implementation of the sample imaging application200. The sample imaging application 200 can include imaging logic 204,atom structure prediction logic 208, atom position detection logic 210,and user interface logic 216. The imaging logic 204 can perform variousfunctions related to acquiring images of a sample. The atom structureprediction logic 208 can perform various functions related to predictingatom structure probability in the acquired images of the sample. Theatom position detection logic 210 can perform various functions relatedto finding positions of atoms in a predicted image including atomstructure probability. The user interface logic 216 can perform variousfunctions related to presenting acquired and enhanced images of a sampleto a user and receiving input from the user. The sample imagingapplication 200 can have other logic components and data structures notspecifically illustrated.

The imaging logic 204 can perform various functions related to acquiringimages of the sample with the charged particle microscope system. Duringnavigation on the sample, the imaging logic 204 can cause the sample tobe placed at different positions relative to the field of view of thecharged particle beam. For example, the imaging logic 204 can providethe controller (112 in FIG. 1 ) of the charged particle microscopesystem with positional information for the sample. In some examples, theimaging logic 204 can derive the positional information for the samplefrom user-defined settings received from the user interface logic 216.The controller can in turn provide control setpoints to the sampleholder (132 in FIG. 1 ) to translate and/or tilt the sample. In somecases, if the charged particle microscope (108 in FIG. 1 ) is mounted toa stage, the charged particle microscope may be translated and/or tiltedrelative to the sample to achieve the desired relative positioningbetween the sample and the scan field of view of the charged particlebeam.

The imaging logic 204 can cause different types of images of the sampleto be captured by the charged particle microscope system. For example,the imaging logic 204 can provide the controller of the charged particlemicroscope system with the detector systems that should be activatedduring imaging of an area of the sample. In one example, the imaginglogic 204 can cause images of the sample to be acquired using onedetector modality (e.g., HAADF detector modality) or at least twodetector modalities (e.g., bright field and dark field detectormodalities or HAADF detector modality and a spectroscopy detectormodality).

The imaging logic 204 can receive detector data from the chargedparticle microscope system. In some examples, the imaging logic 204 canconstruct initial images of the sample from the detector data. Forexample, as the charged particle beam scans across an area of the sample(e.g., in a raster pattern), the charged particle microscope detectorsystem generates an output for each (x, y) scanning beam position andtilt angle of the sample. The detector output for each scanning beamposition can provide information for a pixel of the sample image. Theimaging logic 204 can use the detector output to construct one or moreinitial images of the sample. In some examples, the imaging logic 204can apply time stamps to the initial images constructed and store theinitial images in the data storage (192 in FIG. 1 ) for use by othercomponents of the sample imaging application 200.

The atom structure prediction logic 208 can perform various functionsrelated to predicting atom structure probability in the initial imagesgenerated by the imaging logic 204. In some examples, the initial imagescan have a low SNR. FIG. 3A shows an example of a low SNR image 300acquired from an area of a sample by a HAADF detector. In oneimplementation, the atom structure prediction logic 208 can take one ormore low SNR images, such as the low SNR image 300 shown in FIG. 3A, andpredict atom structure probability in the region of the samplecorresponding to the one or more low SNR images. The output of the atomstructure prediction logic 208 can be an atom structure probabilityimage, which is an image in which there is detectible contrast betweenthe regions of the image likely to contain atom structures and theregions of the image not likely to contain atom structures.

In one example, the atom structure prediction logic 208 predicts atomstructure probability using a trained machine learning model. Forexample, the trained machine learning model can be a neural network(e.g., a convolutional neural network) that has been trained to predictatom structure probability. In one example, the atom structureprediction logic 208 can generate an inference request including a setof one or more initial images (or acquired images). The set of one ormore initial images can form an input vector for a trained neuralnetwork. In some examples, the set of one or more initial images can beimages acquired with the same detector modality. In other examples, theset of one or more initial images can be images acquired with differentdetector modalities. The atom structure prediction logic 208 cantransmit the inference request to an inference engine 220 including atrained machine learning model 224. The inference engine 220 can be inthe same computing environment as the sample imaging application 200 orcan be in a different computing environment (e.g., on an AI server in acloud).

Upon receiving the inference request, the inference engine 220 appliesthe set of one or more initial images in the inference request to theinput of the trained machine learning model 224 to obtain a prediction.The prediction can be an image including atom structure probability.FIG. 3B shows an example of a predicted image 310 outputted by a trainedneural network based on the low SNR image 300 shown in FIG. 3A (theprobable regions of the image 310 containing atom structures have ahigher pixel intensity compared to the regions of the image that are notlikely to contain atom structures). The inference engine 220 generatesan inference including the predicted image. The inference engine 220 canadd other information to the inference, such as a confidence of theprediction of the neural network and/or explanation of the prediction ofthe neural network. The inference engine 220 returns the inference tothe atom structure prediction logic 208.

The confidence of the predictions made by the trained machine learningmodel 224 can be determined using the same techniques that exist for theverification of artificial neural networks. For example, the trainedmachine learning model 224 can be applied to a low quality image (or lowSNR image), and the prediction of the trained machine learning model 224can be compared to a high quality image (or high SNR image) to determinea confidence of the prediction. The low and high quality images can beeither generated with different imaging conditions or simulated byartificially degrading high quality images coming either frommeasurement or from a simulation.

The atom position detection logic 210 can extract atom positions fromthe predicted image including atom structure probability. In oneexample, the atom position detection logic 210 uses image segmentationto extract the atom positions from the atom structure probability image.Various types of image segmentation techniques can be used. Onetechnique can be based on thresholding. In thresholding, a pixelintensity threshold is set for classifying pixels in the atom structureprobability image into atom structure pixel and background pixel. Theatom position detection logic 210 can output the atom positions or canoutput an enhanced image indicating the atom positions. In one example,the enhanced image can be generated by superimposing the atom positionson the predicted image. FIG. 3C shows an example of an enhanced image320 including atom positions, as determined by the atom positiondetection logic 210, superimposed on the predicted image 310 in FIG. 3B(the large dots indicate the atom positions).

The user interface logic 216 can perform various functions related todisplaying a user interface on the display device 188, responding toevents triggered from the user interface, and collecting user input fromthe user interface. For example, the user interface logic 216 canreceive images from the atom structure prediction logic 208 and/or atomposition detection logic 210. The user interface logic 216 can cause agraphical representation of at least a portion of the images to bepresented in a designated area of the user interface. In one example, asan area of the sample is scanned during navigation on the sample orin-situ experiment, the user interface logic 216 can present a sequenceof images in the user interface. The sequence of images can include oneor more acquired images of the area of the sample and one or moreenhanced images generated based on the acquired image(s).

The user interface logic 216 can receive inputs entered at the userinterface by the user. In one example, the inputs can be settings forthe charged particle microscope system or annotations to imagespresented in the user interface. In some cases, the user can select acontrol on the user interface to start imaging of an area of the sample.In this case, the user interface logic 216 can receive the request tostart imaging and transmit the request to the imaging logic 204. Theuser interface logic 216 can cause other information to be displayed inthe user interface, such as one or more metrics related to the operationof the trained machine learning model in the inference engine 220.

In some cases, the sample imaging application 200 can include traininglogic 214, which can generate a request to train or retrain a machinelearning model to predict atom structure probability from low SNRimages. In one example, the training logic 214 can generate a trainingrequest and transmit the training request to a training engine 230 fortraining of a machine learning model 228.

The training engine 230 can receive a training request from the traininglogic 214 and train the machine learning model 228 to perform the taskof predicting atom structure probability in a low SNR image. In someexamples, the training engine 230 can be in the same computingenvironment as the sample imaging application 200. In other examples,the training engine 230 can be in a remote machine learning environmentin a cloud. After training the machine learning model 228, the trainingengine 230 can notify the training logic 214. In response to thenotification, the training logic 214 can deploy the trained machinelearning model 224 (produced by training of the machine learning model228) to the inference engine 220 for use in making predictions.

The sample imaging application 200 can be stored in one or more computerreadable storage media or computer readable storage devices and executedby one or more processors (e.g., processor 180 in the computingenvironment 116 shown in FIG. 1 ). Any of the computer-readable mediaherein can be non-transitory (e.g., volatile memory such as DRAM orSRAM, nonvolatile memory such as magnetic storage, optical storage orthe like) and/or tangible.

Example—Training Dataset for Machine Learning

The training engine 230 can receive a training request from the sampleimaging application 200. The training request can include a trainingdataset 232, or the training engine 230 can retrieve the trainingdataset 232 from a data storage (e.g., data storage 192 in FIG. 1 ), orthe training engine 230 can generate the training dataset 232. Thetraining dataset 232 can include any combination of simulated images,real images, and acquired images of atom structure. Preferably, thetraining dataset 232 contains images from conditions (e.g., noisestatistics and level and atom grid structure) similar to target usecases and materials.

In one example, the training dataset 232 can include low SNR images ofsamples acquired with the charged particle microscope system. Each lowSNR image can be tagged with the name of the sample and the region onthe sample where the low SNR image is obtained. The training dataset canfurther include high SNR images of the samples acquired with the chargedparticle microscope system. Each high SNR image can be tagged with thename of the sample and the region on the sample where the high SNR imageis obtained. Input-output pairs for the training dataset 232 can begenerated by pairing each low SNR image (as input) with one of the highSNR images (as output), using matching sample name and region tags asthe pairing criteria. In some cases, two or more low SNR images acquiredwith different detector modalities (as input) can be paired with one ofthe high SNR images (as output). In some cases, instead of acquiring lowSNR images directly from the charged particle microscope system, theacquired high SNR images can be artificially degraded to form the lowSNR images that are used in the training dataset.

In another example, the training dataset 232 can include simulatedimages of atom structures generated based on the atom structure modeland noise model. A machine learning model can be trained and executed togenerate the simulated images. One example of a technique for generatingsimulated images is described in Lin, R., Zhang, R., Wang, C. et al.TEMImageNet training library and AtomSegNet deep-learning models forhigh-precision atom segmentation, localization, denoising, anddeblurring of atomic-resolution images. Sci Rep 11, 5386 (2021).

In another example, the training dataset can include real imagesobtained from open access electron microscopy datasets, such as theWarwick electron microscopy datasets available atgithub.com/Jeffrey-Ede/datasets/wiki and described in Jeffrey M Ede,“Warwick Electron Microscopy Datasets,” 2020 Mach. Learn.” Sci. Technol.1 045003.

Example—Training of Machine Learning Model

In one example, the training engine 230 trains the machine learningmodel 228 using supervised training cycles interleaved with unsupervisedtraining cycles. In one example, a few supervised cycles (for example,two or more supervised cycles) can be followed by a few unsupervisedcycles (for example, two or more unsupervised cycles).

In one example, the supervised training and unsupervised training canuse a cycle generative adversarial network (CycleGAN) model architecture(see Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros,“Unpaired Image-to-Image Translation using Cycle-Consistent AdversarialNetworks,” arXiv:1703.10593v7, Aug. 24, 2020).

FIG. 6 illustrates a CycleGAN architecture including two generators Gand F. The generator G maps data from domain X to Y. The generator Fmaps data from domain Y to X. The CycleGan architecture includes twodiscriminators Dx and Dy. The discriminator Dy encourages the generatorG to transform X into outputs that are indistinguishable from domain Y.The discriminator Dx encourages the generator F to transform Y intooutputs that are indistinguishable from domain X. The discriminators Dxand Dy are used only during unsupervised training.

The training includes a forward cycle and a backward cycle. In theforward cycle, the generator G takes input x and generates output Y°,and the generator F takes Y° as input and generates output x°. The inputx and output x° should be similar. The difference between x and x° iscycle consistency loss in the forward direction and can be included inoptimization of the network. In the backward cycle, the generator Gtakes input y and generates output X°, and the generator G takes X° asinput and generates output y°. The input y and output y° should besimilar. The difference between y and y° is cycle consistency loss inthe backward direction and can be included in optimization of thenetwork.

The discriminators Dx and Dy are used for unsupervised training. In theforward cycle, the discriminator Dy takes Y° and Y as input and producesan output that signifies real or fake. In the backward cycle, thediscriminator Dx takes X° and X as input and produces an output thatsignifies real or fake. In unsupervised training, the input images x andy are unpaired and are noisy images (e.g., low SNR images). The images Xand Y are real images (e.g., high SNR images).

During supervised training, the discriminators Dx and Dy are not used.The G network is run independently from the large structure using asimulated image as input x and a paired real image as the label y. The Fnetwork is also run independently from the large structure using a realimage as input y and a paired simulated image as label x.

The CycleGAN alternates between the supervised training mode and theunsupervised training mode after a couple of cycles in one of the modes.

The generators and discriminators can be convolutional neural networks.In one example, a U-Net architecture can be used for the generators Gand F. A U-Net is a convolutional neural network that was developed forbiomedical image segmentation (see Olaf Ronneberger et al., “U-NetConvolutional Networks for Biomedical Image Segmentation”,arXiv:1505.04597, May 18, 2015). In one specific example, aparameterized U-Net structure having a 256×256 input with 8multi-resolution layers can be used for the generators. Other types ofconvolutional neural networks, such as VGG architecture or SUNet: SwinTransformer UNet for Image Denoising, can be used for the generators. Inone example, the discriminators can be a PatchGAN discriminator, whichis a convolutional network where an input image is mapped to an N×Narray instead of a single scalar vector.

Example—Sample Imaging Method with Live Assistance

FIG. 4A is a flow diagram illustrating an exemplary method 400 ofimaging a sample with a charged particle microscope system, according toone implementation. FIG. 4B is a block diagram illustrating a portion ofthe method 400. The method 400 can be performed with the system depictedin FIG. 1 and the sample imaging application depicted in FIG. 2 .Operations are illustrated once and each and in a particular order inFIGS. 4A and 4B, but the operations may be reordered and/or repeated asdesired and appropriate (e.g., different operations illustrated asperformed sequentially may be performed in parallel as suitable).

At 410, the method includes navigating to a region of interest (ROI) onthe sample. For example, the sample can be positioned such that the ROIis within a field of view of a charged particle beam outputted by thecharged particle microscope system. The positioning of the sample caninclude transmitting appropriate controls to the sample holder (e.g., bythe controller of the charged particle microscope system) to adjust theposition of the sample relative to the charged particle beam).

At 420, the method includes setting an imaging condition for the sample.The imaging condition can be set by adjusting one or more imagingparameters of the charged particle microscope system. For example, themethod can include setting the dose of the charged particle beam toapply to the sample. If the images are to be produced by scanning theROI, the method can include setting the dwell time or scan speed or scanpattern of the charged particle beam. The method can further includeactivating the detector systems to use in acquiring the image data. Themethod can further include adjusting a tilt angle of the sample. Theparameter settings may come from the sample imaging application and canbe applied by the controller of the charged particle microscope system.

At 430, the method includes acquiring image data from the ROI at theimaging condition set in operation 420. For example, while the chargedparticle beam is incident on the ROI, the activated detector system(s)can collect image data produced from the interaction between the chargedparticle beam and the ROI. In some examples, the ROI can be scanned bysweeping the charged particle beam across the ROI in a raster pattern(or another scan pattern). The activated detector system(s) can capturethe image data as the ROI is scanned. In some examples, image data canbe captured with a single detector modality (e.g., HAADF detectormodality). In other examples, image data can be captured with at leasttwo different detector modalities (e.g., dark field and bright fielddetector modalities or a diffraction detector modality (e.g., HAADFdetector modality) and spectra detector modality (e.g., electron energyloss spectroscopy or energy dispersive X-ray spectroscopy). In otherexamples, a camera can capture images of the sample. The camera caninclude, for example, a CCD (charged-coupled device) imaging sensor, aCMOS (complementary metal-oxide-semiconductor) imaging sensor, or, moregenerally, an array of photodetectors. The camera can be operated in a“movie” mode to capture a sequence of images of the sample.

At 440, the method includes generating a set of one or more initialimages (or acquired images) from the image data acquired in operation430. For example, the image data can include charged particleintensities measured at each scan position of the charged particle beam.Each intensity can correspond to a pixel of the initial image. Theinitial image can be constructed as a set of pixels, where the pixelcoordinates are correlated to the scan position of the beam position andthe pixel value is determined by the measured intensity corresponding tothe scan position of the beam. In some examples, the imaging conditionset in operation 420 results in initial image(s) with a low SNR (i.e.,low SNR image(s)).

At 450, the method includes generating a predicted image indicating atomstructure probability in the ROI using a trained machine learning model.In one example, the trained machine learning model is a neural network(e.g., a convolutional neural network) that has been trained to predictatom structure probability based on one or more low SNR images. Theoutput of the neural network can be a predicted image indicating atomstructure probability. In one example, one or more of the initial imagesgenerated in operation 440 can be applied to the input of the trainedmachine learning model to obtain the predicted image. For illustrativepurposes, FIG. 4B depicts a low SNR image 460 applied to an input of thetrained machine learning model 224, which then generates a predictedimage 464 indicating atom structure probability. In some cases, two ormore low SNR images can be applied to the input of the trained machinelearning model 224 (e.g., two or more low SNR images captured withdifferent detector modalities), and the trained machine learning model224 can output the predicted image based on the two or more low SNRimages.

At 470, the method includes locating atom positions in the predictedimage generated in operation 450. In one example, the atom positions canbe located using image segmentation. Various types of imagesegmentations can be used. One example of an image segmentationtechnique is thresholding. In thresholding, a pixel intensity thresholdis set for classifying pixels in the predicted image into atom pixel andbackground pixel. The method can include generating an enhanced image bysuperimposing atom objects (e.g., geometrical shapes representing atoms)on the predicted image at the atom locations found by the imagesegmentation. For illustrative purposes, FIG. 4B shows the predictedimage 464 inputted to the atom position detection logic 210. FIG. 4Bshows an enhanced image 468 revealing atom locations (the large dots469) in the predicted image 464.

At 480, the method can include presenting images of the sample live on adisplay. In some examples, the method can include recording the imagesfor replaying after the image acquisition. Any combination of the imagesgenerated in operations 440, 450, and 470 can be presented and/orrecorded. FIG. 4B shows that any of the images 460, 464, and 468 can beprovided to the user interface logic 216, which can update the userinterface live with the images. In one example, an initial image (e.g.,a low SNR image) can be presented, followed by a predicted imagegenerated based on the initial image, followed by an enhanced imagegenerated based on the predicted image. In some examples, the predictedimage can be displayed along with confidence of the prediction. Theimages can be presented within a user interface to allow userinteraction with the images. For example, the user may zoom into aportion of the enhanced image and select or tag the portion of the imagefor further investigation. In another example, the user may detect thatthe charged particle beam is not correctly aligned with the atom columnsin the ROI and may determine an adjustment to make to the tilting of thesample based on the images.

At 485, the method can include determining whether to reacquire imagedata from the ROI. For example, the method may determine whether toreacquire image data from the ROI based on user input. For example, ifthe user detects that the sample is not in proper alignment with thecharged particle beam, the user may adjust the setting for the sampletilt angle through the user interface and trigger image datareacquisition of the ROI. In another example, if the user detects thatthe charged particle beam dose is either too small or too large, theuser may adjust the imaging conditions and trigger image datareacquisition of the ROI. If the method determines that the image datais to be reacquired, the method may reacquire the image data byreturning to operation 420 and adjusting the imaging condition (e.g.,based on user input). If the method determines that it is not necessaryto reacquire image data for the ROI, the method can continue tooperation 490.

At 490, the method includes acquiring a final image of the ROI. Thefinal image can be acquired at an imaging condition that is differentfrom the one used in operation 420 to acquire the initial image(s) ofthe ROI. In particular, the imaging condition used in acquiring thefinal image can result in a final image with a higher SNR than theinitial image(s). In some examples, the charged particle beam dose usedin acquiring the final image can be higher than the charged particlebeam dose used in acquiring the initial image(s) and/or the scan speedused in acquiring the final image can be lower than the scan speed usedin acquiring the initial image(s) and/or the scan pattern used inacquiring the low SNR images can be sparser than the scan pattern usedin acquiring the final image.

At 495, the method can include determining whether to navigate toanother ROI on the sample. If the method determines that another ROI onthe sample should be processed, the method can return to operation 410to navigate to the new ROI. In some cases, a user can indicate throughthe user interface that another ROI should be processed. As part ofindicating that another ROI should be processed, the user may specifythe new ROI. Alternatively, the method may automatically determine thenew ROI to process based on a predetermined experiment or navigationplan.

At 499, if the method determines that processing of another ROI is notneeded, image acquisition can be terminated.

Example—Graphical User Interface

FIG. 5 depicts an example graphical user interface (GUI) 500 that may bepresented by the sample imaging application on a display device. A usermay interact with the GUI 500 using any suitable input device and inputtechnique (e.g., movement of a cursor, motion capture, facialrecognition, gesture detection, voice recognition, actuation of buttons,etc.).

The GUI 500 may include a data display region 502, a data analysisregion 504, charged particle microscope control region 506, and asettings region 508. The particular number and arrangement of regionsdepicted in FIG. 5 are simply illustrative, and any number andarrangement of regions, including any desired features, may be includedin a GUI 500.

The data display region 502 may display images generated by the sampleimaging application. For example, the data display region 502 maydisplay any of the low resolution structure images, structure positionimages, structure object images, and structure type images.

The data analysis region 504 may display results of data analysis. Forexample, the data analysis region 504 may display regions of interestindicated by the user in images displayed in the data display region. Insome examples, the data display region 502 and the data analysis region504 may be combined in the GUI 500.

The charged particle microscope control region 506 may include optionsthat allow the user to control the charged particle microscope system.For example, the charged particle microscope control region 506 mayinclude user selectable options that allow parameters of the chargedparticle microscope system to be adjusted.

The settings region 508 may include options that allow the user tocontrol the features and functions of the GUI 500 and/or perform othercomputing operations with respect to the data display region 502 anddata analysis region 502 (e.g., saving data on a storage device).

In view of the above described implementations of the disclosed subjectmatter, this application discloses the additional examples enumeratedbelow. It should be noted that one feature of an example in isolation ormore than one feature of the example taken in combination and,optionally, in combination with one or more features of one or morefurther examples are further examples also falling within the disclosureof this application.

Example 1 is a method of imaging a sample with a charged particlemicroscope system including acquiring one or more first images of aregion of the sample at a first imaging condition with the chargedparticle microscope system; applying the one or more first images to aninput of a trained machine learning model to obtain a predicted imageindicating atom structure probability in the region of the sample; andin response to obtaining the predicted image, causing display of anenhanced image indicating atom locations in the region of the samplebased on the atom structure probability in the predicted image.

Example 2 includes the subject matter of Example 1 and further includesacquiring a second image of the region of the sample at a second imagingcondition with the charged particle microscope system, wherein thesecond imaging condition is selected based on the predicted image or theenhanced image such that the second image has a higher signal-to-noiseratio compared to the one or more first images.

Example 3 includes the subject of Example 2 and further specifies thatacquiring the second image of the region of the sample is triggered fromthe user interface after displaying the enhanced image.

Example 4 includes the subject matter of Example 2 and further specifiesthat a charged particle beam dose in the first imaging condition islower than a charged particle beam dose in the second imaging condition.

Example 5 includes the subject matter of Example 2 and further specifiesthat a first scan pattern used in acquiring the one or more first imagesis sparse compared to a second scan pattern used in acquiring the secondimage.

Example 6 includes the subject matter of any one of Examples 1 to 5 andfurther includes displaying the one or more first images and at leastone of the predicted image and the enhanced image in a sequence on auser interface.

Example 7 includes the subject matter of any one of Examples 1 to 6 andfurther specifies that acquiring the one or more first images of theregion of the sample includes scanning a charged particle beam over theregion of the sample.

Example 8 includes the subject matter of Example 7 and further specifiesthat acquiring the one or more first images of the region of the samplefurther comprises collecting image data from the region of the samplewith a single detector modality and constructing the one or more firstimages from the collected image data.

Example 9 includes the subject matter of Example 8 and further specifiesthat the single detector modality is a high-angle annular dark fielddetector modality.

Example 10 includes the subject matter of Example 7 and furtherspecifies that acquiring the one or more first images of the region ofthe sample includes collecting image data from the region of the samplewith at least two different detector modalities and constructing the oneor more first images from the collected image data.

Example 11 includes the subject matter of Example 10 and furtherspecifies that the at least two different detector modalities comprise adark field detector modality, an annular dark field detector modality, abright field detector modality, an annular dark field detector modality,a high-angle annular dark field detector, a segmented scanningtransmission electron microscopy detector, or an integrated differentialphase contrast detector.

Example 12 includes the subject matter of Example 10 and furtherspecifies that the at least two different detector modalities comprise adiffraction detector modality or a spectra detector modality.

Example 13 includes the subject matter of any one of Examples 1 to 12and further specifies that the trained machine learning model is trainedusing a mixture of supervised learning and unsupervised learning.

Example 14 includes the subject matter of any one of claims 1 to 13 andfurther specifies that the trained machine learning model is trainedusing a cycle generative adversarial network.

Example 15 includes the subject matter of any one of Examples 1 to 14and further specifies that the trained machine learning model includes aconvolutional neural network.

Example 16 includes the subject matter of any one of Examples 1 to 15and further specifies that causing display of the enhanced imageincludes applying image segmentation to the predicted image to find theatom locations.

Example 17 includes the subject matter of Example 16 and furtherspecifies that applying image segmentation to the predicted imageincludes classifying pixels of the predicted image based on a pixelintensity threshold.

Example 18 is a method for scanning a sample with a charged particlemicroscope system including adjusting the sample to different positionsrelative to a field of view of a charged particle beam at differenttimes during an image acquisition; scanning a region of the sample witha charged particle beam at one of the different positions and under afirst imaging condition; acquiring one or more first images of theregion from the scanning under the first imaging condition; applying theone or more first images to an input of a trained machine learning modelto obtain a predicted image indicating atom structure probability in theregion of the sample; responsive to obtaining the predicted image,causing display of an enhanced image indicating atom locations in theregion of the sample based on the atom structure probability in thepredicted image; scanning the region of the sample with the chargedparticle beam at the one of the different positions and under a secondimaging condition that is different from the first imaging condition;and acquiring a second image of the region of the sample from thescanning under the second imaging condition.

Example 19 includes the subject matter of Example 18 and furtherspecifies that the one or more first images have a lower signal-to-noiseratio compared to the second image.

Example 20 is a charged particle microscope support apparatus includingfirst logic to cause a charged particle microscope system to generateone or more first images of a sample having a signal-to-noise ratiobelow a threshold; second logic to apply the one or more first images toan input of a trained machine learning model to generate a predictedimage indicating atom structure probability in the sample; third logicto generate an enhanced image revealing atom locations in the samplebased on the atom structure probability in the predicted image; andfourth logic to cause the charged particle microscope system to generatea second image of the sample having a signal-to-noise ratio above thethreshold.

Example 21 is a system for scanning a sample including a sample holderconfigured to hold a sample; a charged particle source configured toemit a beam of charged particles towards the sample; an optical systemconfigured to cause the beam of charged particles to be incident on thesample; one or more detectors configured to detect charged particles ofthe charged particle beam and/or radiation resultant from the chargedparticle beam being incident on the sample; one or more processors; anda memory storing computer readable instructions that, when executed bythe one or more processors, cause the system to: scan a region of thesample with the charged particle beam during an image acquisition;acquire one or more first images of a region of the sample at a firstimaging condition; apply the one or more first images to an input of atrained machine learning model to obtain a predicted image indicatingatom structure probability in the region of the sample; and causedisplay of an enhanced image revealing atom locations in the region ofthe sample based on the atom structure probability in the predictedimage.

Example 22 includes the subject matter of Example 21 and furtherspecifies that the computer readable instructions, when executed by theone or more processors, further cause the system to acquire a secondimage of the region of the sample at a second imaging condition, whereinthe second imaging condition is selected based on the predicted image orthe enhanced image such that the second image has a highersignal-to-noise ratio compared to the one or more first images.

1. A method of imaging a sample with a charged particle microscopesystem, the method comprising: acquiring one or more first images of aregion of the sample at a first imaging condition with the chargedparticle microscope system; applying the one or more first images to aninput of a trained machine learning model to obtain a predicted imageindicating atom structure probability in the region of the sample; andin response to obtaining the predicted image, causing display of anenhanced image indicating atom locations in the region of the samplebased on the atom structure probability in the predicted image.
 2. Themethod of claim 1, further comprising acquiring a second image of theregion of the sample at a second imaging condition with the chargedparticle microscope system, wherein the second imaging condition isselected based on the predicted image or the enhanced image such thatthe second image has a higher signal-to-noise ratio compared to the oneor more first images.
 3. The method of claim 2, wherein acquiring thesecond image of the region of the sample is triggered from the userinterface after displaying the enhanced image.
 4. The method of claim 2,wherein a charged particle beam dose in the first imaging condition islower than a charged particle beam dose in the second imaging condition.5. The method of claim 2, wherein a first scan pattern used in acquiringthe one or more first images is sparse compared to a second scan patternused in acquiring the second image.
 6. The method of claim 1, furthercomprising displaying the one or more first images and at least one ofthe predicted image and the enhanced image in a sequence on a userinterface.
 7. The method of claim 1, wherein acquiring the one or morefirst images of the region of the sample comprises scanning a chargedparticle beam over the region of the sample.
 8. The method of claim 7,wherein acquiring the one or more first images of the region of thesample further comprises collecting image data from the region of thesample with a single detector modality and constructing the one or morefirst images from the collected image data.
 9. The method of claim 8,wherein the single detector modality is a high-angle annular dark fielddetector modality.
 10. The method of claim 7, wherein acquiring the oneor more first images of the region of the sample further comprisescollecting image data from the region of the sample with at least twodifferent detector modalities and constructing the one or more firstimages from the collected image data.
 11. The method of claim 10,wherein the at least two different detector modalities comprise a darkfield detector modality, an annular dark field detector modality, abright field detector modality, an annular dark field detector modality,a high-angle annular dark field detector, a segmented scanningtransmission electron microscopy detector, or an integrated differentialphase contrast detector.
 12. The method of claim 10, wherein the atleast two different detector modalities comprise a diffraction detectormodality or a spectra detector modality.
 13. The method of claim 1,wherein the trained machine learning model is trained using a mixture ofsupervised learning and unsupervised learning.
 14. The method of claim13, wherein the trained machine learning model is trained using a cyclegenerative adversarial network.
 15. The method of claim 1, wherein thetrained machine learning model comprises a convolutional neural network.16. The method of claim 1, wherein causing display of the enhanced imagecomprises applying image segmentation to the predicted image to find theatom locations.
 17. The method of claim 16, wherein applying imagesegmentation to the predicted image comprises classifying pixels of thepredicted image based on a pixel intensity threshold.
 18. A chargedparticle microscope support apparatus comprising: first logic to cause acharged particle microscope system to generate one or more first imagesof a sample having a signal-to-noise ratio below a threshold; secondlogic to apply the one or more first images to an input of a trainedmachine learning model to generate a predicted image indicating atomstructure probability in the sample; third logic to generate an enhancedimage revealing atom locations in the sample based on the atom structureprobability in the predicted image; and fourth logic to cause thecharged particle microscope system to generate a second image of thesample having a signal-to-noise ratio above the threshold.
 19. A systemfor scanning a sample, the system comprising: a sample holder configuredto hold a sample; a charged particle source configured to emit a beam ofcharged particles towards the sample; an optical system configured tocause the beam of charged particles to be incident on the sample; one ormore detectors configured to detect charged particles of the chargedparticle beam and/or radiation resultant from the charged particle beambeing incident on the sample; one or more processors; and a memorystoring computer readable instructions that, when executed by the one ormore processors, cause the system to: scan a region of the sample withthe charged particle beam during an image acquisition; acquire one ormore first images of a region of the sample at a first imagingcondition; apply the one or more first images to an input of a trainedmachine learning model to obtain a predicted image indicating atomstructure probability in the region of the sample; and cause display ofan enhanced image revealing atom locations in the region of the samplebased on the atom structure probability in the predicted image.
 20. Thesystem of claim 19, wherein the computer readable instructions, whenexecuted by the one or more processors, further cause the system toacquire a second image of the region of the sample at a second imagingcondition, wherein the second imaging condition is selected based on thepredicted image or the enhanced image such that the second image has ahigher signal-to-noise ratio compared to the one or more first images.